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The  Government  Performance  and  Results  Act  (P.L.  103-62),  the  “GPRA”  or 
“Results  Act”  as  it  is  now  termed  (Armey,  1997;  1),  was  passed  more  than  three  years 
ago.  The  Department  of  Defense  (DOD),  like  many  other  agencies,  has  been  engaged  in 
implementation  of  GPRA.  In  this  paper,  I  review  the  actions  the  department  has  taken 
thus  far  to  prepare  to  meet  GPRA  requirements.  I  will  show  that  the  DOD  has  already 
institutionalized  GPRA  strategic  planning  provisions,  and  will  specify  here  some  of  the 
remaining  issues  for  DOD.  I  will  argue  that  these  issues  and  the  influences  of 
performance  budgeting  concepts  on  implementation  have  implications  for  the  putative 
framework  for  the  department’s  Results  Act  implementation,  e.g.,  DOD’s  Planning, 
Programming,  and  Budgeting  System  (PPBS). 


This  paper  is  written  from  the  perspective  of  a  practitioner,  a  participant  observer 
involved  in  the  activities  of  a  DOD  GPRA  performance  pilot  agency  over  the  last  3  years. 
The  subjects  to  be  covered  here  include:  performance  budgeting  from  an  historical 
perspective,  the  legislative  history  of  GPRA,  Office  of  Management  and  Budget  (0MB) 
implementation  guidance  to  the  agencies,  a  chronological  review  of  DOD  implementation 
actions,  and  a  discussion  of  DOD  implementation  issues..  The  primary  methodology 
employed  in  this  paper  is  qualitative  analysis  of  key  documents  and  correspondence  related 
to  the  subject  to  generalize  to  the  emerging  “theory  in  use”  within  DOD  (Patton,  1990), 
and  identify  some  of  the  more  difficult  issues  associated  with  GPRA  implementation.  This 
paper  has  been  developed  as  partial  fulfillment  of  the  graduation  requirements  for  the 
DOD  Senior  Executive  Leadership  Course. 


The  context  of  DOD  actions  at  the  time  this  paper  is  being  written  is  a  “dry  run”  of 
GPRA  implementation  requirements  guided  by  0MB.  The  first  annual  agency  strategic 
plans  required  by  GPRA  must  be  completed  in  September,  1997.  Performance  plans  are 
to  be  submitted  for  FY  1999  (beginning  October  1, 1998),  and  performance  reports  are  to 
be  submitted  annually  to  the  President  and  Congress  beginning  March  1, 2000.  The 
National  Academy  of  Public  Administration  (NAPA),  based  on  an  assessment  of  the  pilot 
agency  performance  plans  by  its  panel  on  Improving  Government  Performance  (NAPA, 
1994;  3),  recommended  that  OMB  and  agencies  should  “begin  as  soon  as  possible...” 
based  on  their  panel’s  findings  that  implementation  was  a  long  and  complex  affair.  In 
response  to  that  recommendation  and  other  similar  views,  OMB’s  review  of  the  FY  1998 
budget  has  been  designed  to  serve  as  a  “dry  run”  for  both  OMB  and  the  agencies;  i.e.,  a 
practice  session  for  the  agencies  and  a  preliminary  look  for  OMB.  The  OMB  reviews  of 
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agency  submissions  during  the  Fall  of  1996  and  Spring  of  1997  are  to  inform  the 
development  of  government- wide  performance  plans. 


GPRA  and  Performance  Budgeting 


The  Results  Act,  in  section  1 1 19,  calls  for  the  establishment  of  performance 
budgeting  pilot  projects.  By  way  of  definition,  the  Act  states  that:  “Such  budgets  shall 
present,  for  one  or  more  of  the  major  functions  and  operations  of  the  agency,  the  varying 
levels  of  performance,  including  outcome-related  performance,  that  would  result  from 
different  budgeted  amounts”  (GPRA,  1993).  There  are,  however,  differing  interpretations 
as  to  what  a  performance  budget  is,  what  it  might  look  like,  and  what  it  might  do.  A 
working  definition  can  be  adopted  as  follows  (Burkhead,  1956;  142): 

A  performance  budget  is  one  which  presents  the  purposes  and  objectives 
for  which  funds  are  required,  the  costs  of  the  programs  proposed  for  achieving 
those  objectives,  and  quantitative  data  measuring  the  accomplishment  and  work 
performed  under  each  program. 


The  committee  reports  on  GPRA  provide  language  which  elaborate  the  statutory 
definition  and  their  intent.  The  Senate  report  is  clear  that  the  desired  result  is  to  receive  a 
budget  “showing  a  direct  relationship  between  proposed  spending  and  expected  results, 
along  with  the  anticipated  effects  of  higher  or  lower  amounts”  (United  States  Senate, 
1993;  18).  The  House  report  goes  further  to  suggest  a  more  complex  approach  where 
“the  budgeted  amount  would  be  changed,  but  various  performance  values  could  be  shifted 
to  reflect  changes  in  priority  or  emphasis....”(United  States  Congress,  1993;  26). 


This  desire  for  performance  data  in  the  Federal  budget  is  not  a  recent 
development:  “In  1913,  the  federal  government’s  Taft  Commission  on  Economy  and 
Efficiency  argued  that  government  officials  needed  to  tie  public  expenditures  to  the 
outcomes  they  produce.”  (Kettl,  1992;  73).  The  basic  question  that  performance 
information  in  the  budget  would  theoretically  answer  for  the  decision  maker  is  best  stated 
as:  “On  what  basis  shall  it  be  decided  to  allocate  x  dollars  to  activity  A  instead  of  activity 
B?”  (Key,  1940;  1 137).  In  addition  to  straight  forward  allocations  within  programs, 
performance-based  goals  and  assessments  of  how  well  these  goals  are  being  achieved  are 
expected  to  assist  the  Congress  in  identifying  cross-program  tradeoffs.  “Such  program 
assessments  should  provide  a  road  map  to  determine  how  limited  federal  tax  dollars  can  be 
applied  most  effectively  in  the  future”  (Armey,  1997;  4). 


The  program  or  account  structure  used  in  federal  budgeting  has  been  characterized 
as  outdated,  and  consequently  a  source  of  limitations  on  the  ability  of  agencies  to  prepare 
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performance  based  budget  requests.  There  are  recommendations  to  modify  the  account 
structure,  and  0MB  has  favorably  entertained  several  agency  proposals  for  revisions  to 
their  program  and  financing  schedules  growing  out  of  agency  adoption  of  goal  structures 
or  as  a  means  for  the  agency  to  align  its  goals  with  its  accounts.  As  another  example,  one 
of  the  supporting  reports  of  the  National  Performance  Review  (NPR)  recommends  the 
creation  of  revised  categories  related  to  GPRA  planning  and  performance  management 
processes  (NPR,  1993;  55). 


The  Federal  budget  is  now  prepared  using  a  line  item  structure,  which  has  been 
described  as  classifying  data  by  organization  (Kettl,  1992;  69)  or  “  around  the  items  to  be 
purchased  ”  (Shick,  1995;  188).  If  one  examines  the  Fiscal  Year  1997  budget  (OMB, 
1996),  the  “Department  of  Defense  -  Military”  account,  for  example,  is  disaggregated  first 
by  use  (personnel,  operation  and  maintenance,  procurement,  research,  military 
construction,  family  housing,  revolving  and  management  funds),  then  typically  by 
organization  (military  personnel  -  Navy,  Special  Operations  Command),  or  item 
(ammunition,  airlift  aircraft).  These  are  generally  inputs  rather  than  outputs  or 
programmatic  outcomes.  The  use  of  even  workload  or  outcome  measures  in  the  budget 
appendix  is  infrequent;  case  workload,  insurance  loss  ratios,  investigations  completed, 
ADA  charges  resolved,  average  cost  per  housing  grant,  tax  fraud  indictment  rate  are  a  few 
random  examples  which  do  appear  in  Appendix  A  of  the  FY  1997  budget.  The  Internal 
Revenue  Service  budget  is  one  of  the  few  agencies  with  extensive  measures  in  the 
appendix  (OMB,  1996;  816).  Of  course,  more  information  is  submitted  with  agency 
budgets  than  appears  in  the  appendix.  A  review  of  four  agency  budget  submissions  found 
386  measures  reported  for  FY  1996,  and  observed  (Newcomer,  1995;  12)  an  increase  in 
the  number  of  measures  reported  of  50%  over  the  prior  year  (FY  1995)  budget. 


The  Office  of  Management  and  Budget  (OMB)  provides  budget  preparation  and 
GPRA  implementation  guidance  to  the  agencies.  The  basic  budget  preparation  guidance  is 
provided  in  OMB  Circular  A-1 1.  Section  34.1  of  the  circular  discusses  the  narrative 
statements  on  program  and  performance  required  in  agency  budget  justifications,  and 
specifies  the  inclusion  of  performance  indicators  (OMB,  1996c;  157): 

The  statements  should  include  appropriate  performance  indicators  and 
goals  agreed  upon  by  the  agency  and  its  OMB  representative  in  their  on-going 
discussions  of  continued  enhancement  of  the  budget  process  through  greater  use  of 
performance  information.  The  statements  can  also  include,  as  appropriate, 
performance  indicators  contained  in  the  agency’s  annual  financial  statements  or 
other  appropriate  indicators. 


The  coverage  of  GPRA-related  guidance  on  such  subjects  as  planning, 
performance  data,  and  GPRA  implementation  generally  has  increased  each  year  over  the 
last  three  years,  culminating  with  the  issuance  of  Part  2  to  A-1 1  in  June,  1996  covering 
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strategic  plans  (0MB,  1996;  237-250).  0MB  had  announced  earlier  its  summer  and  fall 
review  initiatives  in  a  memorandum  to  the  agency  heads  (M-96-22),  the  objectives  of 
which  are  to  continue  to  integrate  GPRA  requirements  into  existing  program 
management  and  budget  processes”  (OMB,  April  1 1,1996).  The  purpose  of  the  summer 
and  fall  reviews  were  described  as  to  “...assure  that  agencies  are  developing  plans  that 
meet  GPRA  requirements,. ..’’for  the  former,  and  as  follows  for  the  latter  (OMB,  April 
11,1996): 


The  second  major  initiative  is  intended  to  produce  consensus  between 
OMB  and  the  agencies  on  the  performance  goals  and  indicators  that  agencies 
would  include  in  their  FY  1999  annual  performance  plans.  This  initiative  will  be 
integrated  with  OMB’s  Fall  Review  of  the  agency  FY  1998  budget  requests. 
Agreement  on  goals  and  indicators  will  help  ensure  that:  (1)  the  performance  plans 
contain  useful  and  relevant  performance  information;  and  (2),  agencies  can  better 
direct  their  measurement  of  current  performance  to  generate  the  baseline  data  from 
which  future  target  levels  would  be  set. 


The  schedule  for  the  Fall  Review  and  related  activities  was  described  in  a 
subsequent  memorandum  (OMB,  1996,  M-96-22,  Supplement  2)  as: 

The  Fall  Review  of  the  proposed  performance  goals  and  indicators  will  be 
conducted  as  a  part  of  the  hearings  and  discussions  with  the  agency  on  its  FY  1998 
budget  request.  These  hearings  and  meetings  generally  occur  between  September 
and  November.  By  December,  1996,  consensus  should  be  reached  either  on  the 
performance  goals  to  be  included  in  the  FY  1999  plan,  or  on  the  schedule  for 
further  work  to  be  done  to  define  these  goals  early  in  CY  1997. 


Defense  Implementation  of  GPRA 


Planning.  Programming  and  Budgeting 


The  Department  of  Defense  not  only  conducts  budget  development  in  the  context 
of  OMB  guidance,  but  also  in  the  context  of  its  own  budget  development  and  resource 
allocation  decision  making  processes.  The  basis  for  these  processes  is  the  Department’s 
Planning,  Programming  and  Budgeting  System  (PPBS).  Introduced  in  1961  by  Secretary 
McNamara,  “The  purpose  of  the  PPBS  is  to  produce  a  plan,  a  program,  and  finally,  a 
budget  for  the  Department  of  Defense”  (DOD,  1984;  1).  Impressed  by  the  discipline  and 
analysis  of  the  PPBS,- President  Johnson  mandated  its  use  in  all  agencies  in  1965.  The 
Nixon  administration,  however,  reversed  that  decision,  probably  because  “PPBS-prepared 
budgets  did  not  fit  either  Congress’s  structure  or  the  way  it  made  budgetary  policy” 
(Kettl,  1992;  77-78),  particularly  in  the  case  of  the  civil  agencies.  The  PPBS  remains  the 
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Department’s  formal  process  for  planning  and  budget  development,  and  has  been 
described  as  a  constantly  evolving  system  (GAO,  1983;  13). 


Some  of  the  groundwork  for  implementation  of  GPRA  came  in  response  to 
passage  of  the  Chief  Financial  Officers  (CFO)  Act;  in  a  memorandum  on  performance 
budgeting,  the  acting  Comptroller  of  DOD  (Comptroller,  1992;  1)  stated: 

In  response  to  congressional  and  0MB  mandates,  and  in  keeping  with  our 
efforts  to  improve  management  practices,  I  am  undertaking  an  initiative  to 
explicitly  include  effectiveness  performance  goals  as  well  as  unit  cost  per  output 
goals  in  operating  budgets.  Effectiveness  performance  goals  are  objective 
indicators  that  describe  the  quality,  timeliness,  and  customer  satisfaction  with  the 
output  of  a  given  activity.  These  goals  will  be  used  to  support  budget  justification 
and  presentation,  and  serve  as  performance  measures  for  the  external  financial 
statements  required  under  the  Chief  Financial  Officers  Act. 


There  is  some  evidence  that  measures  have  been  developed.  The  FY  1995 
Department  of  Army  Annual  Financial  Report,  for  example,  contains  46  non-financial 
measures.  The  Highlights  of  the  Department  of  the  Navy  FY  1998/FY  1999  Biennial 
Budget  contains  a  number  of  performance  measures,  and  an  appendix  lists  the  measures 
under  the  headings  of  the  5  DOD  corporate  goals.  DOD  staff  expect  PPBS  to  evolve 
further  to  incorporate  GPRA  as  it  is  implemented  in  the  Department  (Hamre,  1994;  15). 
Explicit  objectives  for  DOD  implementation  were  outlined  in  a  1995  memorandum 
(OUSD(C),  1995b): 

General  guidelines  for  DOD  implementation  include  that  the  plans  reflect 
the  most  important  aspects  of  Defense;  have  DOD  consensus;  are  integrated  with 
the  DOD  Planning,  Programming,  and  Budgeting  System  (PPBS),  and  are 
supported  by  the  functional  managers  who  will  be  responsible  for  achieving  the 
goals. 


Implementation  Activities 


The  Department  of  Defense  has  been  implementing  the  GPRA  on  two  levels:  on 
one  level,  in  sponsoring  performance  measurement  pilot  activities  under  section  1 1 18;  on  a 
second,  developing  and  securing  internal  consensus  on  DOD-wide  corporate  goals.  The 
first  DOD  GPRA  performance  pilot  designated  ( in  January,  1994)  was  the  Defense 
Logistics  Agency  (OMB,  1994).  Eventually,  a  total  of  seven  DOD  activities  became 
GPRA  pilots; 
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•  Defense  Logistics  Agency 

•  Army  Audit  Agency 

•  Army  Corps  of  Engineers,  Civil  Works 

•  Defense  Commissary  Agency 

•  Air  Combat  Command 

•  Navy  Carrier  Battle  Group  (CINCLANTFLT) 

•  Army  Research  Laboratory 


The  project  to  develop  a  set  of  DOD-wide  corporate  goals  as  part  of  GPRA 
implementation  began  in  1994.  A  departmental  focal  point  for  GPRA  implementation  was 
designated  within  the  Office  of  the  Under  Secretary  of  Defense  (Comptroller), 
Performance  Measures  and  Results  Directorate.  This  is  the  same  organization  that  had 
developed  performance  measurement  criteria  for  DOD  to  assist  in  the  incorporation  of 
indicators  to  the  financial  statements  required  by  the  Chief  Financial  Officer’s  (CFO)  Act 
of  1990,  P.L.  101-576.  A  white  paper  was  prepared  on  corporate  performance  goals  and 
measures  and  circulated  among  the  DOD  staff  for  comment  in  August,  1994,  and 
requesting  designation  of  prospective  working  group  members.  The  white  paper 
summarized  goal  areas  and  existing  measures  from  a  series  of  major  departmental 
documents  such  as  the  National  Security  Strategy  and  the  Defense  Planning  Guidance 
(Comptroller,  1994).  In  January  of  1995,  draft  DOD  mission  and  vision  statements,  and 
seven  corporate  goals  were  provided  to  the  Joint  Chiefs  of  Staff  (JCS)  and  Under/ 
Assistant  Secretaries  of  Defense.  The  working  group  established  the  previous  year  was 
expanded  to  formed  seven  action  teams  (one  for  each  draft  goal).  The  group,  consisting 
of  JCS  and  OSD  staff  representatives,  was  to  develop  performance  measures  to  be  used  in 
the  future  DOD  GPRA  performance  plan  (OUSD(C),  1995a). 


Some  of  the  OSD  staff  representatives  to  the  working  group  were  with 
organizations  that  had  previously  or  concurrently  developed  functional  strategic  or 
performance  plans.  Two  elements  of  the  Office  of  the  Undersecretary  of  Defense 
(Acquisition  and  Technology)  provide  examples.  The  Logistics  Strategic  Plan  (DOD, 
1996a),  for  example,  approved  by  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  in  June,  1996,  was  the  third  logistics  plan  (plans  had  also  been  developed  for 

1994  and  1995).  The  logistics  plan  and  wholesale  supply  system  performance  data  for  FY 

1995  are  both  available  from  the  web  site  for  Materiel  Management  and  Distribution, 
Office  of  the  Deputy  Undersecretary  of  Defense  (Logistics)  at 
http://www.acq.osd.mil/log/mdm.  Similarly,  the  Acquisition  Reform  organization  has 
defined  its  mission  as  consisting  of  7  “musts,”  and  has  made  a  system  for  displaying  key 
metrics  at  3  levels  available  on  its  web  site  at  http://acq.osd.mil/ar. 


One  of  the  major  documents  in  the  PPBS  cycle  is  the  Defense  Planning  Guidance 
(DPG),  a  classified  set  of  assumptions  and  instructions  developed  by  the  Office  of  the 
Assistant  Secretary  of  Defense  (Strategy  and  Requirements)  and  approved  by  the  Deputy 
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Secretary  of  Defense  that  the  Components  follow  in  preparing  their  Program  Objectives 
Memorandum  (POM)  and  budget.  GPRA  requirements  were  first  addressed  in  a  new 
unclassified  Annex  A  to  the  DPG  issued  in  May,  1995.  The  unclassified  annex  discussed 
the  formation  of  the  Comptroller-led  working  group,  and  that  the  next  stage  was  the 
development  of  draft  performance  measures  for  use  in  the  DOD  GPRA  performance  plan. 
Other  correspondence  reflected  similar  content.  The  instructions  to  the  Military  Services 
for  preparation  of  the  Department’s  FY  1997-2001  Program  Objectives  Memorandum 
(POM)  issued  in  February  of  1995  included  the  following  requirement  (OSD(PA&E), 
1995;  8):  “Components  should  review  the  DOD  corporate  level  goals  and  develop 
performance  measures  that  relate  to  them.  During  the  program  review,  components  will 
provide  a  report  to  the  Program  Review  Group  that  identifies  these  performance 
measures.”  This  tasking  was  explained  in  an  April  memorandum  to  the  Military 
Departments  and  Defense  Agency  Directors,  and  a  draft  of  seven  corporate  goals  along 
with  a  strawman  set  of  measures  were  attached. 


The  working  group  was  later  expanded  to  include  Military  Department  and 
Defense  Agency  representatives  (USD(C),  1995b).  The  expanded  working  group  issued 
its  draft  report  in  June,  1995.  In  August  of  1995,  the  working  group  report  was 
distributed  with  a  policy  statement  in  the  transmittal  memorandum  that  the  report  “...is  to 
be  used  as  the  basis  for  GPRA  budget  submissions”  (OUSD(C),  1995c),  presumably  for 
the  FY  1997  budget.  The  budget  exhibits  in  Appendix  A  for  FY  1997,  however,  do  not 
reflect  the  goals  or  measures. 


The  April,  1996,  DPG  also  reflected  increasing  GPRA  implementation  guidance. 
This  time,  the  instructions  for  the  FY  1998-2003  POM  also  contained  a  GPRA  aimex,  but 
it  also  reflected  the  DOD  mission,  vision,  and  the  six  corporate  level  goals  refined  from 
those  developed  the  previous  summer.  In  October,  1996,  the  combination  of  the 
previously  published  goals  5  and  6  was  announced  in  a  program  decision  memorandum 
signed  by  the  Deputy  Secretary  of  Defense.  Corporate  goals  will  appear  in  the  DPG  every 
year  (Maroni,  1996;  23). 


In  addition,  the  National  Military  Strategy  and  Quadrennial  Defense  Review  are 
intended  to  be  used  to  satisfy  the  strategic  planning  requirements  of  the  GPRA  (Maroni, 
1996;  25).  The  National  Military  Strategy  (Joint  Chiefs  of  Staff,  1995)  was  first 
published  as  an  unclassified  document  in  1992.  It  sets  forth  the  general  military  objectives 
of  the  United  States.  The  Quadrennial  Defense  Review  is  described  in  section  923  (the 
subtitle  is  identified  as  the  Military  Force  Structure  Review  Act  of  1996)  of  the  National 
Defense  Authorization  Act  for  FY  1997.  The  purpose  of  the  review  for  1997  is  to 
determine  and  express  “the  defense  strategy  of  the  United  States  ...”  (  US  Congress, 
1997). 
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The  DOD  thus  seems  to  be  positioned  to  comply  with  the  planning  provisions  of 
the  GPRA.  Specific  DOD  mission,  vision,  and  corporate  level  goals  have  been  developed 
and  a  current  published  rendition  appears  below  (Maroni,  1996;  25): 


DOD  Mission,  Vision,  Goals 


DOD  Mission 
Statement 

The  mission  of  the  Department  of  Defense  is  to  support  and  defend 
the  Constitution  of  the  United  States,  to  provide  for  the  common  defense 
of  the  United  States,  its  citizens  and  its  allies,  and  to  protect  and  advance 
U.S.  interests  around  the  world. 

DOD  Vision 
Statement 

The  Department  of  Defense: 

•  Successfully  defends  the  United  States,  its  citizens,  interests  and  allies. 

•  Fields  the  best  trained,  best  equipped,  best  prepared  joint  fighting  force 
in  the  world. 

•  Supports  alliances/security  relationships  that  protect/advance  U.S. 
security  interests. 

•  Advances  national  priorities  in  concert  with  Congress,  other  agencies 
and  the  private  sector. 

•  Serves  as  a  model  of  effective,  efficient  and  innovative  management 
and  leadership. 

DOD  Cornorate- 

1.  Provide  a  flexible,  ready,  and  sustainable  military  force  capable  of 
conducting  joint  operations  to  execute  the  National  Military  Strategy, 
including  the  capability  to: 

•  Rapidly  project  power  to  deter  and,  if  necessary,  fight  and  win  two 
nearly  simultaneous  major  regional  conflicts  (MRCs)  in  concert  with 
regional  allies. 

•  Support  friends  and  allies  and  shape  the  international  environment  in 
ways  favorable  to  U.S.  interests  through  peacetime  overseas  presence. 

•  Conduct  a  wide  range  of  contingency  operations,  including  peace 
operations,  counter-drug  and  counter-terrorist  activities,  noncombatant 
evacuations,  and  disaster  assistance. 

•  Deter,  prevent  and  defend  against  the  effective  use  of  nuclear, 
biological,  and  chemical  weapons  and  their  delivery  means  and  prevent  the 
acquisition  of  such  weapons. 

2.  Recmit  and  retain  well-qualified  military  and  civilian  personnel  and 
provide  them  with  equal  opportunity  and  a  high  quality  of  life. 

3.  Sustain  and  adapt  security  alliances,  enhance  coalition  warfighting,  and 
forge  military  relationships  that  protect  and  advance  U.S.  security  interests. 

4.  Maintain  U.S.  qualitative  superiority  in  key  warfighting  capabilities 
(e.g.,  information  warfare,  logistics). 

5.  Reduce  costs  and  eliminate  unnecessary  expenditures,  while  maintaining 
required  military  capability  across  all  DOD  mission  areas  by  employing  all 
modem  management  tools  and  working  closely  and  effectively  with  other 
government  agencies.  Congress,  and  the  private  sector. 

1996 

Level  Goals 
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Implementation  Issues  for  DOD 

1.  Measures 


The  first  issue  facing  the  Department  concerns  the  selection  and  use  of  appropriate 
performance  measures,  for  use  in  both  strategic  and  performance  planning.  The 
development  of  appropriate  measures  is  a  daunting  task,  one  described  as  taking  years 
(USGAO,  1996;  23),  being  intrinsically  difficult  (Wholey,  1992;  605),  and  requiring  a 
disciplined  approach  to  research  and  development  of  needed  measures  (NAPA,  1994;  18). 
Defense  programs  have  an  added  layer  of  complexity  related  to  the  nature  of  their 
products  (Joyce,  1993;  12): 

National  defense  is  a  particularly  difficult  area  for  which  to  develop 
measures  of  ultimate  performance,  perhaps  because  it  is  almost  impossible  to 
determine,  without  some  military  conflict,  whether  the  objective  of  national 
defense  is  being  achieved.  Measurement  is  further  complicated  by  the  concept,  as 
some  would  argue,  that  the  purpose  of  defense  is  not  winning  wars  but  preventing 
them.  For  this  reason,  many  measures  of  performance  for  national  defense  often 
focus  on  the  quality  and  readiness  of  military  forces,  assuming  that  a  well-qualified, 
appropriately  trained  military  can  best  achieve  whatever  goals  are  paramount  at  the 
moment. 


Measurement  of  military  capability  has  largely  followed  the  models  established  by 
Service  systems  for  measuring  operational  readiness.  Such  models  have  been  criticized  for 
their  indirecmess  and  subjectivity  (Collins,  1994;  6),  and  the  need  to  better  relate  resource 
levels  to  readiness  (Betts,  1995;  88  and  OUSD(P&R),  1994;  ii),  although  the  pilot 
experiences  of  CINCLANTFLT  successfully  used  those  models  as  the  basis  for 
development  of  their  “B”  rating  scheme  . 


0MB  framed  the  issue  of  measures  and  their  relation  to  what  needed  to  be 
improved  in  the  PPBS  concisely  for  the  department  (OMB,  1996): 

We  believe  that  PPBS  provides  an  excellent  foundation  for  the  full 
implementation  of  the  Government  Performance  and  Results  Act.  In  anticipation 
of  the  formulation  of  the  fiscal  year  1998  President’s  budget,  the  Department  is 
encouraged  to  develop  effectiveness  measures  and  conduct  additional  program 
evaluations  of  non-combat-related  and  support  activities.  In  doing  so,  the 
Department  should  focus  on  the  development  of  performance  measures  and 
strategic  plans  that  are  linked  to  resources. 


The  DOD  GPRA  Working  Group  Report  (Comptroller,  1994)  initially  identified 
68  measures  or  types  of  measures  for  use  in  budget  submissions.  Examples  of  some  of 
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these  suggested  measures  include:  unit  status;  mission  capable  rates;  percent  of  required 
strategic  sealift  capability  achieved;  percent  career  retentions;  foreign  military  sales; 
infrastructure  costs;  and,  logistics  response  time.  As  of  July,  1996,  the  goal  and  measure 
structure  proposed  by  the  Directorate  for  Program  Analysis  and  Evaluation  (PA&E) 
consisted  of  the  five  goals  described  in  the  previous  table,  three  quantitative  measures  for 
goal  one,  four  quantitative  measures  for  goal  two,  and  qualitative  criteria  for  goals  three 
through  five  (OUSD(C),  1996;  3).  More  recent  drafts  of  department  goals  have  included 
quantitative  measures  for  goals  3-5.  More  detailed  performance  information  has  been 
developed  within  the  department.  Budget  background  and  supporting  materials  contain 
data  on  force  structure  measures,  flying  hours,  training,  etc.  The  justification  books 
provided  Congress,  however,  have  not  characteristically  contained  such  explicit  data. 


The  sheer  number  of  operational  measures  required  to  manage  an  organization  can 
be  as  daunting  as  the  selection  of  appropriate  aggregate  measures.  The  FY  1996  GPRA 
Performance  Report  prepared  for  the  Commander  in  Chief,  U.  S.  Atlantic  Fleet  covering 
the  George  Washington  Battle  Group  performance  pilot  cited  the  development  and  use  of 
280  performance  indicators  generated  by  1,050  discrete  daily  data  elements  covering  7  of 
their  13  critical  mission  tasks.  The  challenge  at  the  department  level  is  selection  of  those 
measures  at  the  proper  level  of  aggregation,  where  explicit  measures  provide  the  bases  for 
both  meaningful  operational  decisions  and  justification  of  aggregate  program  expenses, 
that  sufficient  performance  detail  is  provided  to  Congressional  committees  to  satisfy  their 
oversight  responsibilities,  and  the  combinations  hopefully  also  pass  a  causal  test  of  reason. 


2.  Performance  Plan 


The  second  issue  concerns  the  format  of  the  DOD  GPRA  performance  report. 
GPRA  performance  planning  pilot  participants  and  their  parent  agencies  took  differing 
approaches  to  satisfying  the  requirement  for  a  performance  plan.  Some  published  separate 
documents.  Others  included  performance  targets  in  their  budget  submissions.  Still  others 
prepared  goal-based  or  performance  budgets  which  integrated  strategic  goals  and 
objectives,  performance  targets,  performance  against  targets,  and  varying  budget  amounts 
associated  with  differing  levels  ofperformance  (CFO  Council,  1997;  3-2).  The 
Department  of  Defense  plans  to  establish  a  summary  set  of  performance  measures  and 
apply  them  during  the  preparation  of  the  Program  Objectives  Memoranda  as  well  as 
during  the  budgeting  process.  As  of  this  writing,  it  is  unclear  what  form  the  “performance 
report”  for  DOD  may  take.  It  seems  most  likely  that  the  performance  plan/report  may  be 
part  of  the  President’s  budget  submission.  However,  process  maps,  instructions,  and 
report  formats  have  apparently  not  been  developed  nor  made  available  to  the  Components 
to  communicate  the  Department’s  approach.  Potential  vehicles  for  die  establishment  of 
requirements  for  the  Components  could  include  the  DOD  Financial  Management 
Regulation,  POM  Preparation  Instructions,  the  Defense  Planning  Guidance,  Program 
Decision  Memoranda,  the  Budget  Estimate  Submission  Guidance,  etc. 
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The  DOD  Financial  Management  Regulation  (DOD,  1996b)  is  issued  by  the 
OUSD  (Comptroller)  under  the  authority  of  DOD  Instruction  7000.14,  “DOD  Financial 
Management  Policy  and  Procedures.”  Chapter  1  of  Volume  2A  provides  basic  guidance 
on  budget  estimates  submission  and  Congressional  presentation.  The  chapter  is  silent  on 
the  use  of  programmatic  performance  measures  in  the  process.  Chapter  3  of  Volume  2 A 
provides  instructions  on  the  preparation  of  input  to  the  President’s  budget  for  the 
Operations  and  Maintenance  appropriation.  These  instructions  (DOD,  1996b;  3-19) 
require  the  submission  of  performance  criteria  on  an  input  format  designated  as  OP-5 
(Part  2).  A  great  deal  of  programmatic  information  can  appear  in  this  input  and  its 
supporting  exhibits.  However,  a  number  of  performance-related  exhibits  are  specifically 
excluded  from  the  Justification  books  provided  to  Congress  (DOD,  1996b;  3-9). 


The  theory  in  use  within  the  department  is  its  PPBS.  Quantitative  performance 
measures  have  been  consistently  applied  in  decision  making,  even  though  many  of  those 
measures  have  been  input  measures  (force  structure  types  of  measures  such  as  number  of 
ships  or  budget  measures  such  as  how  much  is  budgeted  for  weapons  systems 
procurement).  Decisions  on  resource  allocation,  at  least  partially  based  on  performance 
or  departmental  goals,  have  been  made  in  recent  years  during  the  Defense  Management 
Review  Decision  and  Program  Budget  Decision  processes.  In  many  cases,  however,  these 
decisions  have  been  made  at  a  second  tier  in  the  process  and  the  performance  basis  for  the 
decision  was  not  explicit.  The  department  may  need  to  examine  its  current  process 
guidance  for  inclusion  of  the  language  and  templates  necessary  to  make  the  performance 
basis  for  decisions  more  explicit.  Specifically,  the  instructions  in  the  Financial 
Management  Regulation  should  be  updated  so  that  the  budget  products  clearly  reflect 
performance  data  aligned  with  budget  account  data. 


3.  Program  Structure 


The  third  issue  facing  the  Department  is  the  need  or  desirability  of  changes  to  the 
accounting  structure  and/or  processes  used  in  the  PPBS  and  budget  estimate  submissions 
to  0MB.  The  term  “program  activity”  is  defined  in  the  GPRA  as  (GPRA,  1993) :  “...  a 
particular  activity  or  project  as  listed  in  the  program  and  financing  schedules  of  the  annual 
budget  of  the  United  States  Government;....”  0MB  guidance  specifically  links  those 
schedules  with  the  GPRA-required  performance  plans  (0MB,  1994;  2).  Rexibility  is 
given  to  the  agencies  in  the  design  of  this  account  structure  (0MB,  1996d;  3): 

Agencies  wishing  to  change  their  account  structure,  or  modify  program 
activity  listings  in  their  program  and  financing  schedules  to  better  align  these  with 
programs  and  thematic  performance  goals  are  encouraged  to  propose  this  as  part 
of  Fall  Review.  Agencies  may  also  propose  to  shift  financing  for  the  resources 
needed  to  achieve  program  goals  to  these  program  accounts  and  bring  about  a 
better  connection  of  resources  to  results.  Proposing  agencies  should  also  describe 
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their  plans  and  schedules  for  any  consultation  with  appropriate  Congressional 
committees  regarding  these  changes. 


The  decision  to  change  an  agency  account  structure  would  be  based  in  part  on  the 
enhanced  decision-making  potential  represented  by  achievement  of  the  alignment  of 
accounts,  goals,  and  dollar  amounts  as  envisioned  in  a  performance  budgeting  system. 

This  particular  type  of  rational  account  structuring  implies  an  iterative  process  between 
the  structural  decisions  inherent  in  the  structure  of  accounts  and  the  associated  goals  and 
measures,  with  some  of  the  implications  for  impact  on  agency  operations  described  below 
(Kravchuk,  1996;  350): 

Meaningful  measurement  requires  a  rational  program  structure.  Designers 
of  the  measurement  system  should  take  account  of  the  crucial  relationship  between 
program  structure  and  meaningful  measurement.  Multiprogram/  multigoal  systems 
especially  should  be  subjected  to  review,  with  the  intent  to  disentangle  diverse 
programs  where  no  compelling  argument  can  be  offered  for  their  continued 
aggregation.  Development  of  measures  thus  provides  an  appropriate  opportunity 
to  recognize  and  resolve  structural  inconsistencies  that  would  render  measurement 
(and  program  management,  generally)  more  difficult.  Extensive  reorganization  of 
programs  (and  agencies)  is  not  out  of  the  question.  In  this  connection, 
development  of  the  measurement  system  can  be  viewed  as  part  of  a  larger  process 
of  program  restructuring  and  improvement. 


An  account  structure  implies  an  accounting  system  utilizing  that  structure  so  that 
costs  can  be  determined,  at  some  detail,  for  each  product  or  service  of  the  organization. 
Government  accounting  systems  have  not  historically  been  viewed  as  providing  useful 
data.  That  perception  was  one  of  the  drivers  of  passage  of  the  CFO  Act,  which 
established  the  requirement  for  agencies  to  produce  financial  statements  similar  to  those 
used  in  the  private  sector.  One  commentator  noted  that  (Kehoe,  1 995): 

Right  now,  few  government  organizations  have  accurate  cost  information. 
They  know  and  Congress  knows  what  these  organizations  spend,  because  that 
information  is  in  their  budgets.  However,  it  is  very  difficult  to  determine  what  a 
government-produced  product  or  service  actually  costs,  because  federal 
accounting  systems  are  not  set  up  to  provide  accurate  cost  data. 


Current  systems,  however,  are  tenacious.  Changing  such  systems  is  difficult  and 
usually  expensive. 
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4.  Five  Year  Defense  Program  ('FYDP')  Accounts 


Associated  with  the  issue  of  program  structure,  yet  a  distinct  fourth  issue,  is  the 
question  of  changes  to  the  Five  Year  Defense  Program  (FYDP)  accounts.  The  FYDP 
structure  is  used  by  the  Department  to  facilitate  program  review.  The  second  chapter  of 
the  3.5  inch  high  manual  for  this  code  structure  describes  its  purpose  (DOD,  1996;  3): 

In  its  first  dimension,  the  FYDP  is  composed  of  1 1  major  defense  programs 
(6  combat  force-oriented  programs  and  5  support  programs)  used  as  a  basis  for 
internal  DOD  program  review,  and  in  its  second  dimension  by  the  input-oriented 
appropriation  structure  used  by  the  Congress  in  reviewing  budget  requests  and 
enacting  appropriations.  Hence,  it  serves  a  purpose  of  cross-walking  the  internal 
review  structure  which  is  output-oriented  with  the  congressional  review  structure 
which  is  input-oriented. 


The  FYDP  account  stmcture  is  composed  of  program  elements  which  portray  “a 
force  mission  or  a  support  mission  of  the  Department  of  Defense  and  contains  the 
resources  needed  to  achieve  an  objective  or  plan”  (DOD,  1996;  6).  These  missions, 
however,  and  the  codes  themselves,  are  generally  based  on  organizational  entities  rather 
than  on  processes  or  activities.  For  example,  the  program  element  code  0101 127F 
represents  BIB  Squadrons  in  the  Air  Force,  categorized  as  offensive  forces  under 
program  1 ,  Strategic  Forces.  An  example  from  program  2,  General  Purpose  Forces, 
under  unified  commands  would  be  program  element  code  0201 1 120,  Atlantic  Command 
(LANTCOM)  Activities.  Other  elements  have  non-organizational  or  functional  bases,  and 
some  program  expenses  are  tracked  through  this  structure.  Program  element  0602707E, 
for  example,  under  Research  and  Development,  is  titled  Particle  Beam  Technology. 


Within  the  Department,  the  structure  is  maintained  by  the  Program  Budget 
Division  of  the  USD  Comptroller.  The  stakeholders  for  the  structure  include  the  Office  of 
the  Director,  Program  Analysis  and  Evaluation  (responsible  for  evaluation  of  defense 
programs),  and  all  of  the  functional  budget  proponents  (the  Services/ Agencies  and  the 
OSD  Principal  Staff  Assistants). 


Two  types  of  changes  to  the  FYDP  structure  can  be  considered.  The  first  would 
be  driven  by  other  analytical  frameworks  used  within  the  Department.  The  second  would 
be  driven  by  recommendations  for  change  from  various  sources,  including  the  GAO.  The 
primary  example  of  the  first  comes  from  the  activities  and  assessment  methodologies  of 
the  Joint  Chiefs  of  Staff,  developed  through  implementation  of  the  changes  introduced 
under  the  Goldwater-Nichols  DOD  Reorganization  Act  of  1986  and  subsequent  evolution 
of  the  functions  of  the  Joint  Requirements  Oversight  Council  (headed  by  the  Vice 
Chairman  of  the  Joint  Chiefs  of  Staff).  One  of  the  functions  of  the  Council  is  to  oversee 
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the  Joint  Warfighting  Capabilities  Assessments  (JWCA)  process.  This  process  and  the 
inputs  from  the  warfighting  Commander-in-Chiefs  (or  CINCs)  provide  the  bases  for  the 
Chairman’s  Program  Assessment  and  Recommendations,  two  documents  which  impact  the 
Defense  Planning  Guidance.  The  structure  used  in  the  JWCA  process  is  interesting  as 
another  definition  of  the  Department’s  major  business  processes:  “The  Joint  Warfighting 
Capabilities  Assessments  (JWCA’s)  are  a  series  of  mission  area  assessments  which 
investigate  the  capabilities  of  the  armed  forces  in  10  mission  areas.”  ( Keller,  1996;  in.5- 
2).  The  table  below  arrays  the  1 1  major  defense  program  titles  used  in  the  Future  Year 
Defense  Program  alongside  the  10  mission  areas  used  in  the  JWCA  for  ease  of 
comparison. 


Future  Year  Defense  Program 

Joint  Warfighting  Capability  Assessment 

Strategic  Forces 

Strike 

General  Purpose  Forces 

Land  and  Littoral  Warfare 

Command,  Control,  Communications, 
Intelligence  and  Space 

Strategic  Mobility  and  Sustainability 

Mobility  Forces 

Sea,  Air,  and  Space  Superiority 

Guard  and  Reserve  Forces 

Deter/Counter  Proliferation  of  Weapons  of 
Mass  Destruction 

Research  and  Development 

Command  and  Control 

Central  Supply  and  Maintenance 

Information  Warfare 

Training,  Medical,  and  Other  General 
Personnel  Activities 

Intelligence,  Surveillance,  and 

Reconnaissance 

Administration  and  Associated  Activities 

Regional  Engagement/Presence 

Support  of  Other  Nations 

Joint  Readiness 

Special  Operations  Forces 

An  examination  of  the  above  table  show  some  clear  similarities  in  major  categories. 
Mobility  Forces  in  the  FYDP  structure  relate  to  Strategic  Mobility  under  the  JWCA,  for 
instance.  The  differences  are  perhaps  more  noteworthy;  e.g.,  the  JWCA  has  a  separate 
mission  area  for  Information  Warfare.  A  comparison  of  the  structures  would  be  an 
important  necessary  research  project  to  identify  any  rationalizing  changes  to  the  FYDP 
based  on  the  JWCA  structure. 


Examples  of  the  second  type  of  potential  changes  to  the  FYDP  accounting 
structure  include  external  recommendations  on  changes  to  capture  more  effectively  the 
Department’s  infrastructure  in  general,  and  investments  in  information  technology  in 
specific.  The  General  Accounting  Office’s  folio  won  review  of  DOD's  fiscal  year  1996 
FYDP  observed  that  only  some  of  the  DOD  direct  infrastructure  could  currently  be 
identified  (United  States  General  Accounting  Office,  1996;  3): 
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According  to  DOD  officials  within  the  Office  of  Program  Analysis  and 
Evaluation  (PA&E),  DOD’s  efforts  to  identify  and  track  infrastructure  funding 
have  been  underway  for  several  years.  PA&E  officials  told  us  DOD  has  a  better 
understanding  of  the  elements  that  fund  DOD  infrastructure  activities  than  it  had  at 
the  time  of  the  bottom  up  review.  Using  the  FYDP,  DOD  has  clearly  identified 
program  elements  that  fund  infrastructure  activities  and  refer  to  these  as  “direct 
infrastructure.”  However,  there  are  parts  of  the  total  infrastructure  funding  that 
cannot  be  clearly  identified  in  the  FYDP.  According  to  PA&E  officials,  this  is 
about  20  to  25  percent  of  DOD’s  total  infrastructure  funding  and  mostly 
represents  logistics  purchases  which  cannot  be  specifically  identified. 


A  similar  reconunendation  to  expand  the  code  structure  has  surfaced  concerning 
information  technology  (National  Academy  of  Public  Administration,  1996;  23): 

The  increasingly  ubiquitous  nature  of  information  management  has  also 
made  expenditures  in  IM  difficult  to  integrate  into  the  PPBS  cycle.  In  addition, 
program  element  codes,  a  basic  DOD  financial  accounting  building  block,  are  not 
assigned  specifically  to  information  management  elements  of  the  budget.  This 
reduces  the  accuracy  and  credibility  of  the  programs  themselves  and  the 
information  management  costs  associated  with  major  Defense  programs. 


The  greater  potential  for  the  extended  use  of  the  FYDP  account  structure  may  be 
as  a  vehicle  for  aligning  program  expenses  with  program  performance. 


5.  Consultations  with  the  Congress 


The  process  for  consultation  with  Congress  is  the  fifth  open  issue  for  DOD 
because  that  process  is  still  being  defined ,  The  Department  routinely  provides 
information  to  the  Congress;  for  example,  a  Biennial  Budget  Justification  Book  for  the 
FY  1998/1999  President’s  Budget  will  provide  budget  estimate  justification  materials  to  a 
number  of  congressional  committees.  However,  the  level  of  interest  among  Congressional 
members  and  staffs  has  increased  since  passage  of  the  Act  in  1993.  Congress  may  well 
take  an  active  role  in  deciding  what  the  consultation  process  will  be.  Senator  Stevens  sent 
a  memorandum  that  described  the  role  of  Congressional  committees  in  the  effective 
implementation  of  GPRA  as  crucial  (U.S.  Senate,  1996).  A  Washington  Post  article 
noted  that  (Barr,  1997a;  A25): 

Next  week,  house  Majority  Leader  Richard  K.  Armey  (R-Tex)  is  scheduled 
to  testify  before  the  House  Government  Reform  and  Oversight  Committee  to 
signal  that  he  and  House  Speaker  Newt  Gingrich  (R-Ga)  expect  House  committees 
to  promote  the  law,  known  as  GPRA  in  the  government’s  shorthand.  Army, 
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House  Appropriations  Committee  Chairman  Bob  Livingston  (R-La.),  House 
Budget  Chairman  John  R.  Kasich  (R-Ohio),  and  Government  Reform  Chairman 
Dan  Burton  (R-Ind.)  will  help  coordinate  the  House’s  activities,  an  Armey 
spokeswoman  said. 


In  his  testimony,  Mr.  Armey  made  it  clear  that  he  expected  that  consultation  would  occur 
(Armey,  1997;  3-4):  “Starting  now,  and  moving  through  August,  committees  should  be 
working  with  their  executive  branch  departments  to  clarify  what  we  want  from  programs, 
how  that  will  be  accomplished  and  how  we  expect  to  measure  progress.” 


Congressional  pressure  for  meaningful  consultation  is  intensifying.  A  letter  sent  to 
0MB  signed  by  die  House  and  Senate  leadership  is  an  example  of  actions  being  taken  to 
assure  the  Congress  that  “they  have  a  voice  in  the  drafting  of  the  strategic  plans”  (Barr, 
1997b;  A 19)  required  by  September,  1997.  The  department  has  articulated  its  belief  that 
the  National  Defense  Panel  will  provide  a  vehicle  for  consultation  as  a  part  of  the  QDR 
process.  The  new  Secretary  of  Defense,  Mr.  William  S.  Cohen,  committed  to  meetings 
with  several  committees  in  his  confirmation  testimony. 


Conclusions 


Although  the  DOD  has  strong  planning  and  budgeting  processes  in  place,  and  has 
already  adapted  those  processes  to  some  extent  to  incorporate  the  tenets  of  GPRA  (OMB, 
1996),  significant  implementation  issues  remain.  Five  such  issues  are  covered  in  this 
paper.  The  first  is  the  selection  of  appropriate  measures.  The  second  is  the  process  for 
performance  planning  and  reporting,  including  the  crafting  of  guidance  on  that  process. 
The  third  is  the  desirability  of  changes  to  the  program  or  accounting  structure.  The  fourth 
is  the  utility  of  increasing  use  of  the  Five  Year  Defense  Program  account  structure  to  align 
performance  with  expenditures.  The  fifth  is  the  definition  of  the  process  for  consulting 
with  the  Congress.  Some  of  these  discussions  identified  additional  potential  research. 


Two  structural  themes,  I  believe,  will  drive  the  continued  implementation  of  tiie 
Results  Act  in  the  DOD: 

(1)  Requirements  for  external  presentation  of  performance  data  will  force 
some  internal  changes  to  the  PPBS  in  the  near  term. 

(2)  The  intermediate  and  long  term  challenge  will  be  to  develop  ways  to 
align  budget  and  accounting  information  with  the  related  program 
performance  data. 
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One  of  the  specific  purposes  of  the  GPRA  is  to  improve  Federal  program 
effectiveness  and  public  accountability  by  promoting  a  new  focus  on  results,  service 
quality,  and  customer  satisfaction.”  (GPRA,  1993;  2(b)(3).  The  magnitude  of  the 
potential  for  change  through  application  of  a  results-based  management  model  is  hinted  in 
the  GAO  executive  guide  (GAO,  1996;  18): 

Leading  organizations  recognize  that  sound  planning  is  not  enough  to 
ensure  their  success.  An  organization’s  activities,  core  processes,  and  resources 
must  be  aligned  to  support  its  mission  and  help  it  achieve  its  goals.  Such 
organizations  start  by  assessing  the  extent  to  which  their  programs  and  activities 
contribute  to  meeting  their  mission  and  desired  outcomes.  As  the  organizations 
became  more  results-oriented,  they  often  found  it  necessary  to  fundamentally  alter 
activities  and  programs  so  that  they  more  effectively  and  efficiently  produced  the 
services  to  meet  customers’  needs  and  stakeholders’  interests. 


And  as  one  commentator  has  observed:  “It  is  one  thing  to  develop  performance  measures. 
It  is  another,  much  more  significant  action  to  actually  use  performance  measures  to  make 
decisions”  (Newcomer,  1995;  10).  It  will  be  interesting  to  see  whether  future  researchers 
can  find  that  we  passed  this  “use”  test  after  GPRA  implementation. 
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